Story-Making with Improvisational Puppets and Actors

Improvisational actors create engaging vignettes in real time, without detailed planning, and often working within constraints provided by the audience. We are exploring the possibility of creating intelligent computer agents that can be embodied as animated characters, can perform in a manner loosely resembling that of human improvisors, and can tailor their performances to abstract directions offered by users or other system components. We describe implemented systems in two paradigms. Improv Puppets improvise under players’ directions in real time. Improv Actors improvise under scenarios provided in advance. We discuss potential applications related to entertainment, the arts, and children’s learning toys. 1. Directed Improvisation Improvisational actors create their performances in real time, without detailed planning, and often working within constraints provided by the audience. Because improvisational performances are spontaneous, they do not reach for the artistic heights of conventional theater. Nonetheless, audiences find them entertaining. A skilled improv troupe produces stories that are pleasing in form and content. Even when they are only moderately successful, it is amusing to observe the actors work to create a story while meeting the constraints. There is a special pleasure in knowing that each performance is unique. We are exploring the possibility of creating intelligent computer agents that can be embodied as animated characters, can perform in a manner loosely resembling that of human improvisors, and can tailor their performances to abstract directions offered by users or other system components. To illustrate the baisc idea, Figure 1 presents a hypothetical episode involving two agents embodied as animated characters: a large character (Tory) and a small character (Scout). Tory has been directed to act curious and friendly. Scout has been directed to act playful. These directions are quite general; neither character has been directed to do anything in particular, only to behave in ways that reflect the specified moods. Each character incorporates the directions into the “control plan” that will guide his or her own behavior (0) for the duration of the episode. Working within the constraints of the directions, each character improvises his or her own behavior, while simultaneously interpreting and responding to the partner’s behavior as the episode unfolds. At first, Tory is alone in the world. He has many possible behaviors, but he can’t act friendly without a partner. Following his direction to act curious, Tory decides to look around for something (1). Scout enters (2), observes Tory standing still, tries to interpret his behavior, but infers nothing (3). Following her direction to act playful, she decides to start something: to play alone for a while and then hide (4). Observing Scout enter and begin to play alone, Tory tries to interpret her behavior and infers that she is shy (5). Following his direction to act friendly, specialized for a shy character, Tory decides to approach Scout, greet her, and invite her to play (6). 2. Enter 4. Start something Play-alone Hide Time Go-to P Greet Invite-P My Control Plan Time My Control Plan Time Scout's Control Plan Time Tory's Control Plan Enter Play-alone (shy) 5. Observe S (Infer) 6. Interact with shy P 3. Observe T (Infer) Stand still (???) Now Now 1. Look for something Direction: Tory, Act curious & friendly. Direction: Scout, Act playful. 0. Act curious and friendly 0. Act playful Figure 1. Two characters improvise within the constraints of their directions. The box immediately above each character shows its “control plan” for its own behavior. The box above that one shows the control plan a character attributes to its partner, based on abductive inference from the partner’s behavior. Each element in a control plan, numbered 0-6 in order of generation, indicates an “intended” kind of behavior. (The characters shown are embodied as “Woggles,” after the animation developed by J. Bates of CMU.) Although they don't know it, Tory and Scout now have incorrect models of one another’s personalities and behavior and conflicting plans for the interaction. One of them must change, but it doesn't really matter which one. If Tory invites Scout to play first, Scout will drop her own plan and accept his invitation. If Scout hides first, Tory will drop his plan and join her game. Following the most basic rule of improvisation, the charactersaccept all offers [24]. Each one actively seeks to interpret the other’s behavior and readily changes his or her own plans to accommodate the other’s apparent intentions. More generally, we envision these capabilities. Each agent, embodied as an animated character, would accept directions from one or more exogenous sources, either in real time time or in advance of a performance. The directions would constrain, but not completely specify the agent’s behavior. For example, they might assign the agent a role, endow it with personality features, change its mood, or instruct it to perform a kind of behavior. A sequence of directions might shape the narrative structure of the agent’s individual behavior and interactions with other agents. Each agent would construct its own detailed course of behavior, following directions, interacting appropriately with other agents, and filling in unspecified elements along the way. Its improvisations would make sense in the real-time situation, reflect its role, personality, and mood, and manifest certain life-like qualities, such as normal variability and idiosyncrasies in behavior. In some cases, the agent’s improvisations might contribute to the narrative form and content of the story. If a given sequence of directions were repeated on different occasions, the agent might improvise different performances, surprising the audience and perhaps even its director. We believe that improvisational agents, possessing the kinds of capabilities discussed above, would be useful components in a variety of applications, especially those related to entertainment, the arts, and children’s learning toys [4, ]. Applications can be framed in different interaction paradigms, including two we have been studying, “Improv Puppets” and “Improv Actors.” In both paradigms, users and agents collaborate on the creation of simple “stories” performed by the embodied agents. However, users direct Improv Puppets interactively in real time, while they direct Improv Actors with scenarios constructed in advance. As discussed below, these differences also affect other properties of the collaboration and therefore support different kinds of applications. Section 2 describes our technical approach to creating improvisational agents. Sections 3 and 4 describe two implemented systems: “Animated Puppets,” illustrating the Improv Puppets paradigm; and “Master and Servant,” illustrating the Improv Actors paradigm. (Our conference presentation will include videotaped demonstrations of both systems.) Section 5 draws conclusions. 2. Technical Approach 2.1 Logical System Organization An agent has a “body,” a “mind,” and a “mind-body interface” (see Figure 2). An agent’s body is a computer program controlling a graphical manifestation in a virtual world, which it may cohabit with other agents’ bodies. It has sensors and effectors to perceive events and execute motor behaviors in the virtual world. An agent also may have an interface for communicating with users. An agent’s mind is a computer program that performs three basic functions. It integrates perceptual inputs withknowledge and inferences to assess the agent’s dynamic situation. It instantiates and decides when to execute particular behaviors. It performs all processing intervening between situation assessment and behavior. The architecture of the mind is discussed below. An agent’s “mind-body interface” mediates interactions between its mind and body in a tight real-time control loop. It classifies patterns of sensor data as meaningful perceptions, which it relays to the mind. It translates motor commands from the mind into sequences of executable instructions, which it relays to effectors. Thus, the mind-body interface allows an agent’s mind to be transferred among different bodies without modification and vice versa, so long as the agent is given an appropriate set of mappings between sensor data and perceptions and between motor commands and effector instructions. Figure 2 illustrates our logical system organization of these components (see also [9]). This organization permits alternative assignments of system components to processors and to physical locations, as illustrated below. Virtual World Next Behavior Possible Behaviors